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ABSTRACT 

The structure of expertise is investigated along a 
variety of dimensions. A new and distinct approach is used, which is 
based on extensive analysis of theories, models, and practice-based 
strategies found in the literature. This base is used to define a sot 
of dimensions that might be helpful in constructing knowledge 
profiles. The four dimensions designated are: (1) 
cognitive-psychological; (2) educational-psychological; (3) 
psychometric; and (4) content-based. Two approaches afe used to 
support the relevance and validity of the knowledge profile 
dimensions, both based on data from a research project involving 627 
students at the University of Maastricht and the Dutch Open 
University. The extent to which variables along the dimensions give 
information about different components of expertise was examined, and 
the discriminatory power of the knowledge profile dimensions to make 
apparent expertise differences between student subpopulat i ons was 
exT lored. The nonhi erarchi cal dimensions identified suggest that they 
do relate to different components of the prior knowledge state, a 
finding further supported by the profile analyses of the different 
populations. Several knowledge profile dimensions seem promising for 
differentiating prior knowledge. Eight figures and 13 tables present 
analyses data. (Contains 38 references.) (SLD) 
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OTIC RESEARCH REPORTS. 



The Open University is responsible for developing and offering open, higher distance education in which special 
attention is paid to innovations in educational technology. The research in this field is concentrated in "OTIC", that is 
the Centre for Educational Technological Innovations (Onderwijs Technologisch Innovatie Centrum). 
OTIC is also engaged in running projects for other institutions. Here the Centre makes use of OTIC's knowledge and 
experience acquired in research and development. 

The series of OTIC Research Reports consists of publications of the OTIC research projects and aims mainly at an 
audience of fellow researchers. 



THE RESEARCH PROJECT ON EVALUATION AND TEST-FUNCTIONALITIES 

The project "Evaluation and Test-functionalities" focuses on the problems caused by the wide diversity of students and 
the problems with individual and flexible learning-processes of these students. The project leads to an integration of: 

- the results of the project "Prior Knowledge"; 

- the developments of the "Computer Assisted Testing'-project; 

- the developments of "Adaptive Testing" and the IRT-applications (Item-Responds Theory); and 

- the experience of the Open University with the development and use of TSS (Test service systems). 

The main objectives are: (1) to get a discernment of the test and evaluation problems in the open-learning system; (2) 
the generation of the guide-lines, specifications, and technological instruments concerned with the use of prior 
knowledge and experience, flexible testing and the supervision on students during the learning process, and (3) the 
development of instruments which can be useful in solving the given teaching problems. 
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1. Introduction 



In earlier research on domain-specific expertise, we focused on the structure of prior knowledge along the 
content dimension. A series of investigations was conducted within the field of 'economies'. For an 
overview, we refer to Dochy (1992). In this report we investigate the structure of expertise along a variety 
of dimensions. 

Our discussion in relation to the structure of expertise introduces a new approach towards the structure-of- 
knowledge problem. In the theoretical part of this text, we discuss this new and distinct approach which is 
based on extensive analysis of theories, models and practice-based strategies found in literature. This base is 
used to define a set of 'dimensions' that might be helpful to construct 'knowledge profiles'. Four types of 
dimensions are designated : cognitive-psychological dimensions, educational-psychological dimensions, 
psychometrical dimensions and content-based dimensions. 

We provide data to support the relevance and validity of the knowledge profile dimensions. Two approaches 
will be adopted, based on data gathered during a research project involving a sample of university students. 
First, an analysis of the extent to which the variables along the dimensions give information about different 
components of expertise. Second, an analysis of the discriminatory power of the knowledge profile 
dimensions to make apparent expertise differences between a variety of student sub-populations. 
In part 7 and 8 of this report, the first approach is adopted, whereas in part 9, the profile dimensions are 
used to compare a variety of student-populations. In view of the first approach, a distinction is made 
between hierarchical and non-hierarchical dimensions. 

2. Expertise and the 'structure of knowledge' 



The quality and impact of expertise has been a major issue in our research about the role of prior 
knowledge at university level (Dochy 1988a en 1988b; Dochy & Steenbakkers, 1988; Dochy & Bouwens, 
1990; Dochy, Bouwens, Wagemans & Niestadt, 1991; Dochy, 1992). An important conclusion from this 
research body indicates that it looks promising to analyze in more detail the complex of components of 
expertise. A first and promising attempt in this direction focused on the structure of expertise along the 
content dimension (Dochy, Valcke & Wagemans, 1991; Wagemans, Valcke & Dochy, 1991). As is 
argument Ml in Dochy (1992), 'expertise' will be used here as a synonym of 'prior knowledge state'. 
Three important concepts are presented in the former paragraph : expertise, components of expertise and a 
complex of components. These three concepts refer to the value attached to the specific "structure" of 
expertise. Our earlier research revealed that such a structure could be indicated in prior knowledge, e.g. 
along the content dimension. We did for instance discriminate between Optimal Requisite Prior Knowledge 
and Mathematics Knowledge. But it was also suggested that the differentiation of components of expertise 
along other dimensions is needed to be helpful to diagnose and support educational practice (Dochy, Valcke 
& Wagemans, 1991; Wagemans, Valcke & Dochy, 1991). 
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3. The structure of knowledge: a central issue in educational sciences 

The structure of knowledge issue has been dealt with by a variety of theoretical disciplines : cognitive 
psychology, educational psychology, artificial intelligence, etc. From a pragmatic point of view, the issue 
has also been of prime importance in applied sciences like instructional psychology, curriculum development 
theories and psychometry. 

Theoretical disciplines like cognitive psychology, educational psychology, artificial intelligence, etc. have - 
from their points of view - highlighted the "structure of knowledge" resulting in a puzzling variety of 

approaches, focuses, models, theories, research attempts A representative sample of authors comprises 

e.g. Ausubel (1968), de Groot (1946), Mayer (1979), Reigeluth & Stein (1983). We give a short outline of 
some of their specific theoretical contributions : 

- An early, cognitive-theoretical approach appears in the work of Ausubel (1968) who argues that new 
knowledge is only acquired to the extent that it is meaningfully related to existing knowledge. Ausubel 
maintains that knowledge is organized primarily in a hierarchical fashion, which implies that mastery of 
higher knowledge levels assumes mastery of all lower knowledge levels. Additionally, Ausubel advances 
that the various pieces of information integrated within a particular knowledge structure are highly 
interrelated. Thus, the more structured prior knowledge, the more flexible and easy the acquisition of new 
knowledge becomes. 

. Ausubel's conceptualization of learning as the assimilation into prior knowledge, is echoed and extended in 
Mayer's (1979) schema theory. New knowledge is - according to Mayer - assimilated into a hierarchy of 
progressively more specific content within the learner's cognitive store. Thus, the basic learning process can 
be described as the assimilation of new knowledge within hierarchically ordered schemata. 

- Another benchmark is set by the "Elaboration theory". According to this theory, multiple access avenues 
become available to the learner by the activation of alternate relational paths . This is also explained by the 
assumption that knowledge acquisition is facilitated to the extent that information is organized in a 
hierarchically integrated mode (Reigeluth and Stein, 1983). 

These theories are important since they stress the "structured" and hierarchical nature of knowledge, but 
they lack empirical support to ground their validity (Reigeluth and Stein, 1983). Additional support, 
especially for the hierarchical nature of the knowledge organization should be searched for. 

From a rather pragmatic, e.g. instructional-psychological, point of view the structure-of-knowledge 
paradigm should also be investigated in further detail in order to find more efficient ways for using 
instructional technology. Our attempts to find ways to manage prior knowledge indicate e.g. that different 
components of expertise should be taken into account (e.g. along the content dimension) and that 
components of expertise along other dimensions could be helpful to support and diagnose in educational 
settings. 

If we summarize the variety of theoretical and pragmatic approaches, four main types of dimensions to 
structure knowledge can be conceptualized : 



Content related dimensions 
Educational dimensions 
Epistemological dimensions 
Item characteristic dimensions 



In the next parts of this report, these dimensions will be recapitulated and discussed in more detail. But 
more important is our attempt to reveal some new insight into the hierarchical nature of the structure of 
knowledge. As indicated earlier this will be the main issue in the empirical part of this text. 
It should also be noted that - when putting forward these theories and approaches towards prior knowledge - 
we adopt an information processing view towards learning v see Sternberg, 1985a & 1985b). This is inspired 
by our dynamic approach towards knowledge acquisition; a departure which is also advocated by 
information processing theory. 



8 



Validation of knowledge profiles dimensions 



p.3 



4. Towards a new key concept : knowledge profiles 



As such, the concept of 'knowledge profiles' is not found in literature. Only 'student profiles' (Wolf, et. 
al., 1991) and 'cognitive profile' (Letteri, 1980) have some similarity in meaning. 

Our use of the concept 'profile' is derived from the practice, common in educational research, of plotting as 
a graph (or profile) the raw or standardized scores of a group or individual (Keeves, 1988). In analyzing 
research data, comparisons are made between persons or groups of persons in terms of a set of 
measurements about related aspects. For each person or group a profile can be obtained by combining the 
results of the set of parameters. The comparison between individual profiles is known by the generic term 
'profile analysis'. Consequently, the concept 'knowledge profile' is used when achievement scores (e.g. on 
a prior knowledge test) are analysed following the parameters along a dimension and when the results of <iiis 
analysis are plotted in a graph. 




Figure 1: Example of a profile 

Figure 1 shows the relationships between some key concepts. A "dimension" is the basis to construct a 
knowledge profile. Each dimension represents an approach towards the structure of knowledge since it 
introduces a related set of expertise components. These components are named "parameters". 

As suggested above, our concept "knowledge profiles" is related - to a certain degree - to Letteri's use of 
the concept. His work at the Center for Cognitive Studies, University of Vermont, focuses on the 
development of an individual's cognitive profile. A profile in Letteri's conception is based on a continuum 
along which a variety of cognitive dimensions are put one next to the other. Letteri combines up to seven 
dimensions such as scanning (focus), breadth of categorization and cognitive complexity. An individual's 
cognitive profile is the diagram, the picture that results after positioning individual scores in relation to each 
of these dimensions along the continuum (Letteri et. al., 1982). 

According to Letteri and Kuntz (1982) very high correlations between an individual's cognitive profile and 
its performance on intellectual tasks, the ability to learn and school performance hs.,e been detected. The 
results of the Letteri studies are at least amazing. His cognitive profiles can for instance help to separate 
seventh and eight grade students into significantly different achievement levels; can account for as high as 
87 percent of the variance in posttest scores; and predict those scores at a level of .05 or better. 
Moreover the results show that a cognitive profile is a basic determinant of academic achievement and can 
accurately help to identify specific learning deficits that contribute significantly to low academic 
achievement. Cognitive profiles seem to be reliable predictors of low/high academic achievement (Letteri, 
19°0). 

Although the Vermont studies provide evidence for the potential of cognitive profiles, some critical remarks 
are needed. 

First, the work at Vermont concentrates rather on 'cognitive styles' and 'characteristics of cognitive 
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functioning*. This is particularly obvious if we look in more detail to the dimensions used to construct the 
cognitive profiles. So there is a clear distinction between their and our approach. Our knowledge profile 
dimensions clearly consist of parameters referring to structure-aspects of knowledge. Second, the Letteri 
profiles seem not to be appropriate for adult learners because they are based on theories and research 
concerning child development. 

Third, we perceive profiles as diagrams based o>\ a single dimension along which the parameters are clearly 
interrelated. The use of a set of such dimensions results in the construction of multiple knowledge profiles 
(i.e. one for each dimension). Letteri constructs only one profile, based on a variety of dimensions. His 
major focus is on the correlation between this profile and school performance. As a consequence, the 
remedial power of bis profiles remains restricted. The Letteri profile is therefore to be considered as an 
instrument to differentiate groups performing below or above average, taking into account the perceived 
correlations between the relative positioning on the profile and external measures of school performance. 
His remedial method is thus not based on the specific overall profile structure but on related external 
measures of school performance. 

In contrast, our multi-profile approach generates "profile analysis", an application of multivariate analysis of 
variance (MANOVA) in which several dependent variables are measured on the same scale (Tabachnick & 
Fidell, 1989). This profile analysis can provide relevant information with diagnostic and remedial value. In 
this way, the profiles help us to identify learning deficits to be remediated. 



5. Overview of the theoretical dimensions 



In this paragraph, we review a representative sample of dimensions and parameters currently found in 
literature. Each of these dimensions is based on a specific model or theory of knowledge structuring, which 
will only be discussed in short. Literature references are supplied to document the reader, looking for more 
information. As suggested in part 3 of this text, the "structure of knowledge" issue has been dealt wrth from 
a large variety of viewpoints, resulting in a quartet of dimensions. 

A first group of dimensions is classified according to cojnmon models of the economics domain. Other sets 
of dimensions are based or theories of knowledge representation, knowledge structure, learning theories, 
text representation models and psychometric theory. 

A first question in relation to each dimension is whether they are applicable as structures to find components 
of expertise. Secondly - and this will be discussed in the empirical part of this text - are these dimensions 
useful to differentiate groups of students. In this way we can scrutinize the descriptive, explanatory and 
remedial prospect of our knowledge profiles. 

5.1 Content Dimensions 

5.1.1 Economics Subdomains Dimension 

Content is one of the most exercised dimensions in order to categorize domain knowledge. Classification 
based on the parameter 'subdomain* means that the domain of economics will be divided in regular subject 
matters within the science of economics. One possibility, as implemented in the curriculum of the University 
of Maastricht, contains nine parameters. 



1. Repotting 

2. Financing 

3. Organization 

4. Marketing 

5. Macroeconomics 

6. Microeconomics 

7. Public finances 

8. International economic affairs 

9. Behavioural and social sciences 
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5.1.2 Curriculum Dimension 

Some parts of the content of a science are supposed to be mastered by the students at certain moments 
during their study. These moments are called the curriculum levels (first and second year). A team of 
economists have helped to define these levels. These curriculum levels are subsequent, but too broad to be 
supposed hierarchical. 



1. First year level 

2. Second year level 



5.1.3 Curriculum Accent Dimension 

Within economics it is common to differentiate between two main streams, representing a different accent, 
i.e. general economics and business administration on the one hand and quantitative economics on the other 
hand. 



1 . General economics and business administration 

2. Quantitative economics 



5.2 Cognitive Psychological Dimensions 
5.2. 1 Propositional Dimension 

Knowledge, representation as used in schema theories (Dochy & Bouwens, 1990) takes certain propositions 
or nodes rs a starting point. A proposition is the smallest unit that can be qualified as true or false. 
According to most schema theories there are five kinds of nodes: Physical State (PS, statement that refers to 
an ongoing state in the physical or social world), Physical Event (PE, statement that refers to a state change 
in the physical or social world), Internal State (IS, statement that refers to an ongoing state of knowledge, 
attitude, or belief in a character), Internal Event (IE, refers to a state change in knowledge, attiiude or belief 
in a character), Goal (G, statement that refers to an achieved or unachieved state that a person wants) and 
Style (S, statement that refers to details about the style or manner in which an action or event occurred. 
Further examples and elaborations of these parameters are given in Dochy and Bouwens (1990). These 
nodes are used in the representational theory of Graesser (1981) to represent knowledge as a network, of 
labelled statement nodes that are interrelated by directed arcs (see further). As such, the nodes do not have 
any hierarchical relationship. 



1. PS 

2. PE 

3. IS 

4. IE 

5. O 

6. S 



5.2.2 Node Relation Dimension 

The "Node Relation Dimension" is based on characteristics of the interrelations between the propositions 
presented above. Relations between propositions can be classified as node relation or arc parameters: 
Reason (R, a Goal node is a reason for another Goal node), Initiate (I, a State or Event initiates another 
Goal node), Consequence (C, a State, Event or Goal node that has the consequence of another State or 
Event node), Manner (M, an Event or Goal node occurs with some style), Property (P, a person, object or 
entity has some property that is a State node) (see also Dochy & Bouwens, 1990). 
These arc parameters are not of a hierarchical nature. The overview on the next page shows what specific 
relations between the propositions have been identified. 
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l.G - G REASON 



2. PS - G . INITIATE 
IS-G 
PE-G 
IE-G 



3. PS - PE CONSEQUENCE 
IS-PE 
PE-PE 
IE-PE 
G - PE 
PS-PS 
IS - PS 
PE-PS 
IE-PS 
G -PS 



4.PE-S/G MANNER 
IE-S/G 
GE-S/G 



5. PS PS PROPERTY 



5.2.3. Cognitive Complexity Dimension 

Mc Daniel (1991) proposes five levels of cognitive complexity. These parameters are designed to measure 
thinking processes by determining the cognitive complexity apparent in written interpretations of complex 
situations. 

Level 1: Unilateral Descriptions 

The situation is simplified. It focuses on one idea or argument. Alternatives are not identified. No new 
information, meaning, or perspectives are brought in. "Good-bad" and "either-or" assertions are made. 
Appeals to authority or simple rules. Information is simply paraphrased, restated or repeated. 

Lt-vel 2: Simplistic Alternatives 

Simple and obvious conflicts are identified, but the conflicts are not pursued or analyzed. Develops a 
position by dismissing or ignoring one alternative and supporting the other with assertions and simple 
explanations rather than through deeper assessment of the situation. 

Level 3: Emergent Complexity 

More than one possible explanation or perspective is identified. Complexity is established and preserved. 
New elements are introduced. Supports position through comparisons and simple causal sta. nnents. 

Level 4: Broad Interpretations 

Broad ideas help define and interpret the situation. Ideas within the perspective established are manipulated. 
There is a clearly recognizable explanatory theme. Ideas are integrated into "subassemblies" each supporting 
a component of the explanation. 

Level 5: Integrated Analysis 

The situation is restructured or reconceptualized and the problem is approached from a new point of view. 
A network of cause-and-effect relationships is constructed. Ideas are Integrated and extrapolated. Arrives at 
new interpretations by analogy, application of principles and generalizations. An organizing framework is 
constructed, connections are given and consequences are predicted. 
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1. Unilateral deieriptiorus 

2. Simplistic alternative* 

3. Emergent complexity 

4. Broad interpretations 

5. Integrated analyiii 



5.3 Educational-Psychological Dimensions 

Discussing the first two dimensions of this type, i.e. the behavioral and the content dimension, it is 
noteworthy to mention Component Display Theory (CDT, Merrill, 1983) which makes use of related 
concepts. CDT can be described as a set of prescriptive relationships that can be used to guide the design 
and the development of learning activities. A basic assumption of CDT is that there is more than one type 
of learning and more than one kind of memory structure. Primary aspects of CDT are objectives, learning 
activities and tests. According to CDT, all objectives or test items can be classified in cells of a matrix, 
based on a content and a performance dimension. The CDT content dimension distinguishes 'facts, 
concepts, principles and procedures'. The CDT performanca dimension differentiates 'remember, use and 
find'. This is also conform to GagnS's tripartite : 'information, skills and strategies'. CDT holds that this 
performance-content matrix can be considered as a taxonomy, thus suggesting a hierarchical base for the 
two detennining dimensions in the matrix. 

5.3.1 Behavioral Dimension 

The much used distinction between 'declarative and procedural knowledge' can be further operationalized as 
'to know, to understand, and to apply'. These three concepts are considered as equivalent to 'recognition, 
reproduction and production' (De Corte, et al., 1976). The concepts can also be related to the classification 
: Appreciation, recognition and reproduction of information (= declarative) or production or applications 
(interpretative, convergent, divergent or evaluative production) which can be viewed as procedural (Keeves, 
1988). The three parameters do also correspond with the basic taxonomic levels proposed by several 
educationalists such as Bloom, Guilford, De Corte and De Block (see Keeves, 1988). Most researchers 
agree that these parameters are of a hierarchical nature which has also been supported by empirical evidence 
(Keeves, 1988). Research suggests also that there is some justification for treating Bloom's lower levels as 
being taxonomic. This should not be the case for the levels 'synthesis' and 'evaluation'(Madaus, et al., 
1973). 



1. Know 


1. Recognition, 


1. Appreciation, 


1. Declarative 




Reproduction 


recognition, 




2. Understand 




and reproduction 








■ of information 




3. Apply 


2. Production 


2. Interpretative 


2. Procedural 






convergent, 








divergent or 








evaluative 








production 





5.3.2 Content Dimension 

At the content level we can differentiate five parameters: facts, concepts, relations, structures and methods. 
This is in accordance with the work of Guilford where he distinguishes several product parameters, and the 
work of other authors (Keeves, 1988). ihese parameters are widely accepted as being hierarchical (Keeves, 
1988). 



1. Facta 

2. Concepts 

3. Relations 

4. Structures 

5. Methods 
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5.3.3 Epistemological Dimension 

Based on the levels of knowledge representation of Brachman and Schmolze (1985), five levels can be 
differentiated along this dimension. These parameters can also be considered as the most appropriate 
combinations of behaviour- and content parameters, as clarified between brackets : knowledge identification 
(identifying facts and concepts), knowledge conceptualisation (insight in concepts), epistemological analysis 
(to know and understand relations and structures), logical analysis (to know and understand methods), 
implementational analysis (application of methods). These levels are considered as hierarchical since they 
are a combination of the hierarchical behaviour and content dimension. 



1. Knowledge identification 

2. Knowledge conceptualisation 

3. Epistemological analysis 

4. Logical analysis 

5. Implementational analysis 



5.3.4 Layers of Knowledge Dimension 

A difference can be made between: static knowledge (description of concepts and relations), knowledge of 
different types of inferences, knowledge representing elementary tasks (procedures), and strategic 
knowledge (Clancey, 1983). 

The first layer contains the static knowledge of the domain: domain concepts, relations and complex 
structures, such as models of processes or devices. 

The second layer is the inference layer. In this layer we describe what inferences can be made on the basis 
of the knowledge in the static layer. Two types of entities are represented at the inference layer: meta- 
classes and knowledge sources. Meta-classes describe the role domain-concepts can play in reasoning. For 
example, a domain concept like 'infection* can play the role of a finding in a consultation process, but it 
may also play the role of hypothesis. Knowleuge sources describe what types of inferences can be made on 
the basis of the relations in the domain layer. Examples are the specification and the generalisation of 
knowledge sources, which both make use of a subsumption relation in the domain layer. 
The third layer is called the task layer. At this level the basic objects become goals and tasks. Tasks are 
ways in which knowledge sources can be combined to achieve a particular goal. 
The fourth layer is the strategic layer in which knowledge resides which allows a system to make plans - 
i.e. create a task structure - to control and monitor the execution of tasks, to diagnose when something goes 
wrong and to repair impasses. 



1. Static knowledge layer 

2. Inference layer 

3. Task layer 

4. Strategic layer 



5.4 Item Characteristics Dimensions 

Although these dimensions are of a completely different nature, they are in the context of our research 
purposes of importance. Items are used to measure the mastery of (prior) knowledge. Moreover, test items 
are clues to the activation of prior knowledge. The way the individual is instigated to show his mastery of 
knowledge and the way certain knowledge is (represented to the learner can also be related to the "structure 
of knowledge issue". 
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5-4. 1 Number of Propositions Dimension 

A proposition is the smallest unit that can be considered as a separate statement that can be judged as true 
or false. In schema theories (Dochy & Bouwens, 1990), propositions or nodes have a central function in the 
structure of schema. It is assumed that the amount of propositions determines the degree of structure needed 
to answer the item correctly. According to the mean amount of propositions in our items, we found that 
most items had 4 to 9 propositions. Three parameters have been identified in relation to this dimension : 



1. < 5 propositions 

2. > 4 < 10 propositions 

3. > 9 propositions 



5.4.2 Information Level Dimension 

The stem of an item is the general information which is given and which must not be evaluated as true or 
false. This correct information proceeds the question(s) for which this information should be taken into 
account. A stem can be connected to one or more subsequent questions. Therefore, the spatial and logical 
distance between the general information part of an item and the question part of the whole item is larger 
than for simple items without stem. 



1 . Kerns with a stem 

2. Items without a stem 



5.4.3 Representation Level Dimension 

Adhering the representation structure, used in the research of Boekaerts (1979), i.e. visual, verbal and 
symbolic representation, we used four parameters along this dimension. These parameters are also closely 
related to the four content levels in Guilford's structure of intellect model: figural, symbolic, semantic (the 
verbal factor) and behavioral (nonverbal information) and the Twyman (1985) categories : verbal, pictorial 
and schematic. Since test-items are always - in part - based on a textual representation of information, our 
dimension distinguishes only between parameters that are combinations of knowledge representation. 



1 • Textual-graphical 

2. Textual 

3. Textual-schematic 

4. Textual-symbolic 



5.5 Hierarchical and Non-hierarchical Dimensions. 

Of importance in this theoretical discussion of the dimensions for the construction of knowledge profiles is 
the hierarchical or non-hierarchical nature of the dimensions. Empirical validation of the dimensions and 
their further application, has to take this particularity into account. Table 1 on the next page presents a 
summary. An asterix (*) indicates fhat the dimension is considered to be of a hierarchical nature. 
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Table 1 : Hierarchical and non-hierarchical dimensions 



Profile dimensions 




Economics subdomains dimension 




Curriculum dimension 




Curriculum accent dimension 




Prepositional dimension 




Node relation dimension 




Cognitive complexity dimension 




Behavioral dimension 




Content dimension 




Epistemological dimension 




Layers of knowledge dimension 


* 


Number of propositions dimension 




Information level dimension 




Representation level dimension 


9 
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6,. Looking for empirical evidence : 
data collection, data screening end general validation procedure 

6.1 Data Collection 

When looking for empirical evidence to support the theoretical assumptions and theoretical constructs in the 
former part of this text, research data were gathered from an experimental sample. A test - consisting of 
154 items and called the Domain-Specific Knowledge State Test (DS KST) - was adrninistered to 627 
students of the University of Maastricht and the Dutch Open university, starting studying "economics". 

Construction of the domain-specific knowledge state test (DS KST) 

There is evidence that learning is much more domain-specific that earlier learning theorists believed (Dochy, 
1992). It is obvious that domain-referenced testing provides a reasonable possibility of measuring an 
individual's knowledge status and of tracking his progress within a certain domain. Apart from that main 
reason, there is more evidence to account for increasing application of Domain-Specific KS Tests. 

First, it can be assumed that the learning process is also influenced by prior knowledge that is broader 
than strictly subject-specific prior knowledge. For this reason, a domain-specific test was developed 
covering the whole domain (up to a certain degree of difficulty of specialization). In our case we are 
concerned with the domain of economics. 

Second, differences between students concerning specific subjects are sometimes rather large. In this 
case, a test at beginners level would not be able to bring to light all of the differences between the students. 
The chances of doing this is greater when using a test related to end terms, i.e. one whose level corresponds 
to the end of the second year university study. 

Third, because some students have already gained a great deal of experience in their working 
environment or have already attained a relatively high educational level (some higher vocational education 
degree or a university degree), a test set at beginner's level (final VWO level) would not be capable of 
measuring part of the prior knowledge state. 

Fourth, the recent trend known as 'open learning' which uses flexibility as a key concept, is strongly 
related to the use of DS tests. Open learning tries to take the students' prior knowledge into account and 
allows them to study at their own place and pace. Students have a large degree of freedom in choosing 
educational media and objectives. A sudden openness related to objectives and other choices needs an 
appropriate assessment instrument. Domain-specific tests seem to be appropriate. Other instruments rather 
neglect the primary conditions of open learning. DS tests not only reveal measurement results, but also 
information and guidelines to attack deficiencies. Testing should serve the learning process. 

A fifth argument is the psychometric quality of the item bank for DS tests. The amount of available items 
for a course is often restricted and does not allow the removal of items on the basis of insufficient 
psychometric quality (validity, reliability, p-value). This problem disappears when using the larger item 
banks for DS tests. 

A sixth reason in favour of DS tests is the possibility of using them for different functions, i.e. 
assessment of entrance level, of progress and certification. Further, it allows to focus on different 
dimensions that exceed beyond the content level. 

Finally, there is the trend towards internationalization of higher education and co-operation between 
European universities. In this respect, DS tests enable comparisons of individual students and comparisons 
of institutions to be made. 

For these reasons, a domain-specific test was developed. This test, which is aimed at the whole domain of 
economics, is set at the level which should be attained by the end of the second year of university study. 
The heterogeneity of the test population (or student population) is so great that a test at beginner's level 
would not be able to bring to light all of the differences between the students. After all, it can be assumed 
that students with years' of working experience in, for instance, the financial sector, or students who have 
obtained other academic (WO) or higher vocational education (HBO) diplomas will have advanced further 
than the beginners level in certain areas, and may achieve a score approximating to the final 'economist' 
level. In other words, because some students have already gained a great deal of experience in a working 
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environment or have already attained a relatively high educational level, a test set at beginner's level (final 
VWO level) would not be able to measure some of the prior knowledge state. 

The University of Limburg possesses a weald of c .perience in constructing tests, especially tests 
associated with end terms. Our DS Knowledge State Test was constructed as a representative random test of 
items selected from the item bank of the Economics Faculty of the University of Limburg. The items in the 
data bank are classified in 9 subject areas, viz.: reporting, financing, organization, marketing, macro- 
economics, micro- 
economics, public finances, international economic affairs and behavioral and social sciences. 

If test analysis is to be quick and the method of answering the items is to be simple, multiple-choice or 
true/false items must be used. This test consists of multiple-choice questions which can be answered with 
true/false or ?. The ?-al tentative is taken as a third alternative in order to prevent guessing. 
There is no problem in relation to validity since the test clearly represents - to a very large extent - the 
domain and has been developed by a team of domain experts. The psychometric quality problem is 
especially in game when determining the reliability of the test. If we calculate the alpha-coefficient, the test 
can be considered as very reliable: alpha =.93. This high reliability is probably caused by the fact that the 
test is long (154 items), which gives a high alpha-coefficient. Moreover, the test is not necessarily 
homogeneous which implies that basic assumptions on which the calculations of the alpha-coefficient are 
based could have been violated. 

Table 2: Alpha-coefficients for the parameters of the economics subdomains and curriculum accent dimension 
and mean alpha-coefficients 



Parameters 


oxoefficierrt 


N items 


can 


Economics subdomains dimension 








Reporting 

Finance 

Organisation 

Marketing 

Macro-economy 

Micro-economy 

Public 

International Economics 
Behavioral & Social Sciences 


.57 
.64 
.69 
.63 
.71 
.74 
.51 
.55 
.63 


18 
18 
18 
18 
25 
25 
11 
11 
10 


.63 


Curriculum accent dimension 








General economics 


.93 


139 


.69 


Quantitative economics 


.45 


15 





One solution to this problem might be to check the reliability of sub-parts of the test, making use of the 
knowledge profile dimensions. Calculation of alpha was repeated for two of these dimensions (economics 
sub-domains and curriculum accent), in order to be able to present a mean reliability score. When 
reorganizing the test into more homogeneous sub-parts, the alpha-coefficient and a mean alpha-coefficient 
were calculated. The results of this procedure are summarized in table 2. To enable the figures to be judged 
in a better perspective, the number of items within each sub-group of items is also given. Mean alpha seems 
to be > .63. This reliability score is - taking into account the restricted number of items in certain sub-parts 
of the test - acceptable for our research purposes. 

The 154 items were analyzed - separately - by the members of the research group. In reviewing the items, 
the researchers attempted to classify each item along each of the 12 dimensions, discussed in part 5 of this 
report. 
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Table 3 : Applying profile dimensions - success or failure 



Economics Subtlo mains Dimension 


+ 


i^umcuium Dimension 




Curriculum Accent Dimension 


+ 


Prepositional Parameters Dimension 




Node Relation Dimension 


+ 


Cognitive Complexity Dimension 




Behavioral Dimension 


+ 


Content Dimension 


i 

+ 


Epistcmological Dimension 


+ 


Layers of Knowledge Dimension 




Number of Propositions Dimension 


+ 


Information Level Dimension 


+ 


Representation Level Dimension 


+ 



In relation to three dimensions the researchers encountered too many difficulties (table 3) : 

The "Prepositional Parameters" dimension could not be applied since all items consist of 
more than one node. But this dimension was helpful as a base to determine the "Node 
Relation" parameter of an item. 

The parameters along the "Cognitive Complexity" dimension were C >o vague and not 
defined at an operational level to be applied consistently. Moreover, they implied a 
reformulation of the items, which was not acceptable. 

The "Layers of Knowledge" dimension was felt to be a duplicate of the "Behavioral Level" 
dimension. 

In relation to the other dimensions an inter-rater reliability was obtained > .8". If there was discussion in 
relation to the categorization of a specific item along a dimension, discussion resulted in a consensus on the 
final classification of the item. 

6.2 Data Screening 

After screening and reorganizing the items, the raw item scores were used as the base for calculating new 
subscores for each parameter along each dimension. To be able to compare the parameter subscores, mean 
% scores were calculated. This helped to define the data set in table 4. This table presents an overview of 
the mean % scores, the standard deviation and the minimum and maximum score in relation to each 
parameter on each dimension. The last column indicates the number of test items on which the calculation 
of the parameter scores is based. In association to most parameters, the score is based on a sufficient 
number of test items. Only the "node relation" and the "curriculum accent" dimension present some 
problems since the numbers of items is not sufficiently equilibrated over all the parameters of these 
dimensions. 
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Table 4 : Descriptive data of all the parameters for each dimension 





m 


a 


twin. 


max. 


n 


ECONOMICS 


42.85 


19.14 


.00 


100.00 


154 


REPORT 


25.63 


14.69 


.00 


83.33 


18 


FINANCE 


26.37 


ic « 


.00 


72.22 


18 


ORGAN 


34.50 


18.58 


loo 


83.33 


18 


MARKET 


35.74 


16.80 


.00 


83.33 


18 


ma nun 


?K fj. 


IH.Df 


.00 


72. GO 


25 


MICRO 


24.44 


14.89 


.00 


80.00 


25 


PUBLIC 


32.93 


18.50 


.00 


90.91 


11 


f UTCD4I1T 


OA. 01 

CU«7 1 


17 OO 


on 


81 fO 


1 1 


BEHAV 


17.38 


17.80 


.00 


80.00 


10 


LEVEL 1 


30.64 


12.63 


.00 


65.69 


102 


LEVEL2 


22.31 


15.91 


.00 


78.85 


52 


HUM 1 


LfiJO 




on 


73 33 


15 


GENERAL 


28.20 


12.80 


.00 


66.91 


139 


REASON 


27.91 


34.42 


.00 


100.0 


2 


INITIATE 


20.45 


18.17 


.00 


87.50 


8 


CONSEQ 


29.09 


13.21 


.00 


72.41 


58 


MANNER 


39.39 


23.77 


.00 


100.00 


5 


PROPERTY 


26.93 


12.65 


.00 


69.14 


81 


KNOW 


29.82 


15.37 


.00 


79.17 


24 


i n«iun i 




13 38 


00 


71 .72 


99 


APPLY 


22.67 


11.96 


.00 


70i97 


31 


FACTUAL 


22.43 


19.62 


.00 


100.00 


6 


CONCEPT 


35.25 


16.67 


.00 


80.95 


21 


RELAT 


30.84 


14.57 


.00 


75.00 


32 


CTBI IPT 




13 5? 


00 

tUV 


78 57 


5A 

JO 


SKILL 


24.11 


12.92 


.00 


76.92 


39 


KIDENT 


29.22 


16.18 


.00 


80.00 


15 


K CONCEPT 


36.75 


19 85 

1 T.O«7 


.00 


90.00 


10 


EPISTO 


28.02 


13.39 


.00 


72.62 


84 


LOGICAL 


30.27 


17.67 


.00 


86.67 


15 


IMPLEM 


22.40 


11.89 


.00 


70.00 


30 


< 5 


27.18 


12.63 


.00 


67.07 


G2 


> 4 < 10 


13.63 


6.08 


.00 


31.00 


44 


> 9 


6.93 


4.18 


.00 


22.00 


28 


UITHS 


29.92 


13.45 


.00 


68.00 


105 


WITHOUT 


39.32 


19.67 


.00 


100.00 


49 


TEXT 


28.77 


13.25 


.00 


67.68 


99 


CONCRET 


36.58 


20.71 


.00 


88.89 


9 


SCHEME 


28.71 


16.26 


.00 


71.43 


14 


SYMBOL 


22.47 


12.68 


.00 


68.75 


32 



Considering the data in the table, some preliminary conclusions can be drawn in relation to the value of the 
dimensions and parameters to specify c joponents of expertise : 

The mean % scores for the specific parameters along a dimension show remarkable 

differences. 

Also the or values vary to a large extent, even when excluding the values based on a too 
small number of items (last column). 

The high tr-values indicate that the score distribution does not represent a normal 
distribution of scores. The fact that the results are based on a prior knowledge state test 
are responsible for this large score-distribution. 

The fact that for all parameters/dimensions the minimum value is 0, indicates that the test 
has been difficult for the students. On the other hand, some student are able to show their 
mastery of economics by obtaining relatively high mean % scores. 
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Further screening of the data reveals : 
There are no missing date. 

The large standard deviations, mentioned above, might imply that the distribution of the 
scores is not normal (Skewness, Kurtosis) and that the distributions are influenced by 
outliers. The Kurtosis-values vary between -.637 and .989. Skewness-values vary between 
.080 and .881. Both measure suggest acceptable distributions of the data. To check multi- 
variate normality, box-plots of the data have been screened. They reveal there are 
extremes and outliers, but their number remains restricted. Moreover, outliers are expected 
(given the fact we measure mastery of expertise) and considered part of the particular 
distribution of our data. They are properly part of the population from which we intend to 
sample. 

Analysis of the correlation matrix reveals high r-values, but never > .8 which implies that 
all distinctive variables are non-redundant and do not measure comparable issues'. Multi- 
collinearity is therefore not a problem. 

Homogeneity of variance-covariance matrices is a necessary assumption if we compare the 
subscores along the different dimensions. Calculation of the Cochrans C and Bartlett-Box F 
and their respective significance-levels, indicate that this assumption is not violated. 

6.3 Validation Procedure 

When validating the different knowledge profile dimensions, two approaches will be adopted : 
an analysis of the extent to which the variables along the dimensions give information 
about different components of expertise; 

an analysis of the discriminatory power of the dimensions to make apparent expertise 

differences between student sub-populations. 
In part 7 and 8 of this text, the first approach is adopted, whereas in part 9, the profile dimensions are used 
to compare a variety of student-subpopulations. In view of the first approach, a distinction is to be made 
between hierarchical and non-hierarchical dimensions. Hierarchical dimensions imply consecutive 
intercorrel&tions between the variables. This can be evaluated by applying multiple linear regression 
techniques. Non-hierarchical dimensions imply low correlations between the dimension variables. This is to 
be evaluated by analysis of the correlation matrix. 



7. The value of the non-hierarchical dimensions 



7. 1 Economics Subdomains Dimension 

As described in part 5. 1 of this report, the economics-subtopics reflect a rather practical subdivision of the 
economics domain in view of educational purposes. It is therefore to be foreseen that the mean % scores for 
the different economics subjects do correlate to a certain extent (all r-coefficients are statistically significant 
at the 1 % level) 2 , since this subdivision does not (and cannot) reflect an absolute and objective split-up of 
the economics domain. 



1 Tabachnik & Fidell (1990, p.87) use a limit of r > .9 to distinguish redundant variables. 

1 When indicating significance levels in the next part of this text ** indicates p < .01 and * indicates p < .05. 
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Table 5 : Correlation Matrix subtopic-dimension 



H 1 1 2 I 3 j 4 | s 1 6 | 7 | g | 1~[ 



1. Report 


1.00 


JIB 


381 


556 


.427 


.47$ 


377 


JS0 


.442 


2. FTntnfn 




1.00 


.683 


.«m 


352 


327 


309 


303 


318 


3. Ofgm 






1X0 




.492 


.483 


.472 


.446 


.483 


4. Mulct 








IJCO 


.543 


319 


.497 


.479 


300 


5. Mtcro 










U» 


.662 


367 


sn 


314 


6. Miero 












1.00 


.481 


.549 


.520 


7. Pi±Ue 














1.00 


.562 


.456 


8* lulcnui* 
















1.00 


.474 


9.Bebn>. 


















1.00 



The data in table 5 confirm our expectations. The different economics subtopics are intercorrelated. If we 
concentrate only on the r-coefficients > .6, we get the following picture : 



Corr. coo11. r > .8 
1-fl 



0.9- 



0.8- 




I 



Roport Flnano Organ Market 



Maoro 



Mloro 



M Report 


\M Finance 


VZA Organ B 


■1 Markat 


Hi Maoro 


IZ3 Mloro 1 



Figure 2 : Correlation structure Economics subdomains dimension 

The six following subtopics are highly intercorrelated ; "Report, Finance, Organ, Market, Macro & Micro". 
This can be due to high degree of content-links between these economics subdomains or the fact that 
mastery of a particular subdomain explicitly builds on the mastery of another domain. This is particularly 
important in relation to the subdomain "Finance" and "Marketing". 

If we execute a linear regression analysis 3 by entering the mean % scores of the subdomains "Report, 
Macro and Organ" stepwise in the regression equation, we find that up to 50.3 % of the variance in the 
"Finance" is accounted for by these three subdomains. 

In a comparable way, stepwise regression from "Organ & Finance" on "Marketing" reveals that up 51.6 % 
of the variance in the "Marketing" mean score is accounted for by the scores for "Organ & Finance". This 
interdependence can be taken into account when setting up learning activities or guidance initiatives for 
students. 

An alternative interpretation of the correlation coefficients links the mastery of the different economics 
subdomains to the previous experiences of students with the particular subtopics in their secondary 
education. A subpopulation of the starting university students might have got an introduction to certain 
economics domains, such as micro- and macro-economics and reporting (accountancy). The subsequent high 



» If regression analysis is executed, also the residuals are analyzed in order to detect violations of regression assumptions (linearity, 
normality of the distribution of the dependent variable, constant variance). If violations are detected, this will be reported. 
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scores for these subtopicp in the expertise test are therefore expected to correlate to a certain extent. 

7.2 Curriculum Dimension 

In the prior knowledge test, the test items were subdivided into two course level sets : "Level 1" grouping 
items that evaluate mastery at the first year level and "Level2" that evaluate the mastery at the second year 
level. 

As table 4 shows, the mastery of both the levell (m = 30.64) and ihe level2 (m= 22.31) items is restricted. 
As expected, the "level2" total subscore is the lowest indicating the higher difficulty level of this set of 
items. Content experts indicate that mastery of levell items is to a certain degree related to mastery of 
"level2" items since part of the "level2" course content builds on the mastery of "levell" items. This is 
confirmed by linear regression analysis. This analysis indicates that the "level2" mean % score of the 
students helps to explain 38.2 % of the variance in the "level2" mean % scores 4 . 

7.3 Curriculum Accent Dimension 

A generally accepted subdivision in economics is splitting up the domain into general economics and 
quantitative economics. Our expertise test, does only reflect this subdivision to a very limited extent since 
only 15 items can be classified as quantitative economics questions whereas 139 items concentrate on the 
mastery of general economics knowledge. It is therefore predictable that the test results along this dimension 
will not be very useful. Analysis of the mean scores (m,^ = 24.36; m totK ^= 28.20) and standard 
deviations = 13.77; = 12.80) in table 4 indicate that the students master the two types of 
economics knowledge to a comparable extent. Moreover, the intercorrelation between both measure is rather 
high (r =.6197"*). The subdivision between quantitative and general economics is therefore not able - to a 
sufficient extent - to separate knowledge components. 

7.4 Node Relation Dimension 

The correlation matrix on the next page clearly indicates that most parameters along this dimension can be 
clearly separated from each other. The "property" parameter is highly correlated with the "initiate" (.6428**) 
and the "consequence" (.8523*") parameter. 



Corr. coon. 
-1 -rt 




Figure 3 : Node Relation Dimension - Correlation structure 



* When executing linear regression analysts, basic assumptions have been checked. No violations have been detected, for instance t'-« 
distribution of the residuals is normal. 



9 

ERIC 

ummmsiMiM 



Validation of knowledge profiles dimensions 



7.18 



7.5 Number of Propositions Dimension 

Since the number of propositions in test items can be considered as a measure comparable to difficulty 
levels, a gradual decrease in the mean scores for the three variables on this dimension is expected. 
The data in table 6 confirm our expectations (m prop0 , 1 =27.18; 111^=13.63; m pn]p< ^=6.93). Analysis of the 
correlation matrix reveals striking results (all r-coeff. are statistically significant, p < .001): 

Table 6 : Correlation matrix : Number of propositions 



I 


1 


2 


• 1 


1 1. < 5 propositions 


1.00 


.765 


.803 ] 


1 2. > 4 < 10 propositions 




1.00 


.761 


1 3. > 9 propositions 






1.00 1 



The results in table 6 can be interpreted as follows : students who are able to solve items, consisting of > 9 
propositions (3) are able to solve items consisting of a number of proposition > 5 < 10 (2) and are 
certainly able of solving items consisting of < 5 propositions (1). This is confirmed by a linear regression 
analysis : the scores for items with > 9 propositions explain 64.3 % of the variance in the scores of items 
with < 5 propositions. If we add the scores for items with > 4 < 9 propositions to the regression 
equation, up to 70 % of the variance is accounted for. 



Corr. coeff. (r > .6) 



y 




Figure 4 : Intcrcorrelation between mean % scores for "Number of Propositions" 
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7.6 Information Level Dimension 

As explained in the theoretical part of this text, items with a stem have in common that the spatial and 
logical distance between the basic information and the question part of the item is greater than in items 
without a stem. To a certain extent, this dimensions can be compared to the "number of propositions 
dimension". 

It is expected that items with a stem are more difficult than those without a stem. This is confirmed by the 
data in table 4 when comparing the mean % scores and a = 29.92; m^ = 39.32; o^.^ - 

13.45; ownta,, = 19.67). This is also confirmed by the analysis of the correlation matrix. Both variables are 
highly correlated (r = .8246"). This can be explained as follows : the students being able to solve items 
with a stem, are to a very high extent able to solve items without a stem. 

Linear regression confirms this hypothesis : the mean % scores for the items with a stem help to explain 68 
% of the variance in the scores for items without a stem. 

8. Validating hierarchical dimensions 

Up to 5 dimensions have been identified as being "hierarchical" (table 1). As explained in part 6.3 
hierarchical dimensions imply consecutive intercorrelations between the parameters along the dimensions. 
This can be evaluated by applying multiple linear regression techniques. Beforehand, some additional 
remarks are to be made because we isolate profile dimensions from the economics subdomain dimension. In 
validating the hierarchical nature of the dimensions abstraction is therefore made of the subject matter on 
which the items are based. There is as a consequence no control on interaction effects between the 
dimensions and the economics subdomains dimension. If e.g. items evaluating the mastery of "micro- 
economics" at the "insight" level are compared to items evaluating the mastery of "finance" at "application" 
level and the latter economics subdomain is much more complex and difficult than the former, the analysis 
results in relation to the hierarchical structure of "insight" and "apply" along the behavioral dimension might 
be disturbed. 



8.1 Behavioral Dimension 



Oorr. ooelf. 




Figure 5 : Behavioral Dimension - Correlation structure 



Figure 4 clearly indicates features of the hierarchical nature of this dimension, although the highest 
correlations are observed in relation to items measuring the mastery of items at insight-level (insight-apply 
= .72246"; insight-know = .78633"). The latter fact is consistent with earlier validations of this kind of 
dimension (the taxonomic classification of Bloom : knowledge - comprehension - analysis - synthesis and 
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evaluation). Keeves (1988, p.346) refers e.g. to a validation by Ebel, Hill & Horn. 

The three lowest levels, "knowledge, comprehension and application", are comparable to our three first 
behavioral levels and are found to be well-hierarchical. Higher up in the hierarchy a branching takes place. 
If we extend our analysis to a regression analysis in which we evaluate the interrelation between the three 
consecutive behavioral levels, we get the following picture : 



Table 7 : Regression analysis behavioral dimension 



Independent variable 


Dependent variable 


% variance explained 


Know 


Insight 


61 % 


Know*, Insight 


Apply 


52.2 % 



When entering the parameters "know" and "insight" in the regression equation, the contribution of the 
parameter "know" is considered too low 5 and therefore excluded from the regression equation. 
The results in table 7 help to confirm the assumptions about the hierarchical nature of this dimension since 
the preceding pirameter(s) always help to explain the major part of the variance in the mean % scores for 
the subsequent parameter along the dimension. 



8.2 Content Dimension 



Oorr. coeft. 

i -n 




Figure 6 : Content Dimension - Correlation structure 

In the picture it can clearly be seen that the correlation coefficients increase along the consecutive 
parameters of this dimension. This can be considered as a first indicator to support the hierarchical nature of 
this dimension. Only in relation to the parameter "methods" there is a minor decrease in the r-values. 



5 The limit of PIN =.05 is reached. 
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Table 8 : Regression analysis content dimension 



Independent variables 


Dependent variables 


% variance explained 


(F)acts 


Concepts 


15.5% F 


(F)*ctB - (Qoncepts 


Relations 


47.9% C F 


(F)acU - (C)oncepts - (Relations 


Structures 


71.0% RCF 


(F)acts - (Qoncepts - (R)elations - (S)truetutes 


Methods 


59.6% S F C R 



The data in table 8 are very consistent. At the consecutive levels, the hierarchy between "facts - concepts - 
relation and structures" is respected. In the second column thu letters and the order in the letters refer to the 
pattern in which the independent variables help to explain the variance in the dependent variable. This order 
always respects the supposed hierarchy. 

Only at the skills-level, there is a deviant structure. This suggests that "skills" might be of a more general 
and complex nature than "facts, concepts, relations and structures". 



8.3 Epistemological Dimension 



As explained earlier, this dimension can be considered as a combination of the behavioral (8. 1) and content 
dimension (8.2). The structure is especially based on the assumption that the lower behavioral parameters 
are rather linked with lower content parameters (e.g. knowledge of a fact) and that the higher behavioral 
parameters are rather linked with the higher content parameters (e.g. application of a theory). 
This mixture of two hierarchical dimensions might impose difficulties in terms of their interaction. The 
correlation matrix is e.g. not helpful to recognize - at first sight - the hierarchical structure. 



Corr. coetr. 



aa-ir 




Figure 7 : Epistemological Dimension - Correlation structure 

There are some high correlations between the lower and higher parameters along this dimension. But 
nevertheless, the highest correlations are observed when the distance between a high and another parameter 
along the dimension is small (implem-logical) and the lowest correlations are observed when the distance 
between a high and another parameter is large (e.g. "implem - kident). 

A regression analysis in which the parameters are evaluated in terms of their explanatory power for the 
consecutive parameters gives us the following picture : 
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Table 9 : Regression analysis - epistemological dimension 



Independent variables 


Dependent variable 


% cxpl. variance 


(Ki)dent 


(Kc)oncept 


29.7 % 


(Ki)dent - (Kc)onccpt 


(E)pisto 


56% KiKc 


(Ki)dent - (Kc)onoept - (E)pisto 


(L)ogical 


56 % EKi 


(KQdent - (Kc)oncept - (H)pisto - (L)ogical 


(I)mplem 


50.7 % EL 



The regression analysis confirms the former preliminary conclusions. The hierarchy is disturbed but still 
present. The proportion of the variance in the consecutive parameters is always explained by the former 
parameters with the closest distance along the dimension. Moreover, the proportion of explained variance i; 
> 50 %. Lower level parameters are sometimes excluded from the regression equation. 

8.4 Representation Level Dimension 



Corr. ooelf. 



0.8 -f 




Taxtual-araphloal TSxtual Taxtual-aohamstlo Tfcxtual-aymnollo 



Figure 8 : Representation Level Dimension - Correlation structure 

Figure 7 reveals high intercorrelations between all parameters and the "text" parameter. This is to be 
expected since the test items are never based solely on concrete, schematic or symbolic information. The 
items are a priori based on textual information which is supported, enhanced, enriched or supplemented with 
information of another nature. Nevertheless it is remarkable in table 4 that the mean % scores for the. 
subsequent parameters decrease from 36.58 % to 22.47 %, suggesting higher difficulty levels for the items 
based on higher order representation levels. 



Table 10: Regression analysis representation level dimension 



| Independent variables 


Dependent variable 


% expl. variance | 


| (Qoncret 


(T)ext 


33.0 % 1 


| (Qoncret - (Dext 


(Sc)heme 


51.0 % T 


| (Qoncret - (Dcxt - (Sc)heme 


(Sy)mbol 


51.6 %TC ~| 



Table 10 confirms the expected disturbance of the hierarchical nature by the predominance of the textual 
representation mode in most items. The "text" parameter is always the parameter responsible for the highest 
proportion of the variance in the subsequent parameter. 
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9. The discriminatory power of the knowledge profile dimensions 

To validate the 10 knowledge profile dimensions in an additional way, their power to make explicit 
differences in expertise between subpopulations of students has been determined. Three subdivisions in 
student populations have been researched : 

- The differences) between students studying in differing university contexts. 

- The differences) between students with low and high levels of expertise. 

- The differences) between economics and law students, studying the same course. 

A more detailed description of these three investigations can be found in Dochy & Valcke (1992), Valcke & 
Dochy (1992) and Wagemans, Valcke & Dochy (1992). 

9. 1 The differences) between students studying in differing university contexts. 

Students have - in the Dutch context - the option to choose for a variety of university contexts and 
educational approaches at university level (e.g. problem centred approach, experiential learning, distance 
education, etc). A relatively new development (since 1985) in this perspective is the provision of "open and 
distance university education" by the Dutch Open university (Ou). The question can be put forward whether 
this new university setting is just another higher education institution enriching the variety of already 
existing provisions or whether the Open university answers the need of (a) specific student populations); 
e.g. second chance, older students, female students, handicapped people, foreign students, post-university 
students ? A way to look for answers to this question is to analyze - by interviews, questionnaires, etc. - 
demographic variables of the actual student population of the Ou. 

Another approach goes beyond these surface variables and analyzes in more detail the expertise of the 
students opting for the Ou. The research question, which results from this approach is whether the expertise 
of the students, opting for this study context is different from students studying at regular universities. 
Making use of the 10 knowledge profile dimensions, an extensive analysis has been executed on research 
data of students studying economics at the Dutch Open university (Ou) and students studying economics at 
the University of Maastricht (RL) where a problem centred approach towards education is adopted. 

Although the overall prior knowledge mean % score for the economics domain is not significantly different 
between the Ou- and the RL-student population, a univariate analysis of the knowledge profiles helps to 
enlighten obvious and significant differences in the complex composition of components of prior knowledge. 
These results are confirmed and reinforced by the results of the profile analysis (multivariate analysis of 
variance). The results of the profile analysis (parallelism test and discriminant analysis) help to reveal 
specific and significant differences between the profiles of both student populations. Up to seven of the ten 
knowledge profile dimensions prove to be of relevance (table 11). Especially the "content dimension" and 
the "epistemological dimension" were helpful to describe these differences. 

A further extension of our profile analysis (flatness test) helps to induce further evidence to support the 
validity of the knowledge profile dimensions, since all profiles are non-flat. This implies that all dimensions 
are helpful to identify a specific structure in the mastery of expertise. Following this structure, the mastery 
of certain components of expertise is better than for other components. 
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Table 11 : Profile analysis data for the parallelism test 



Profile Dimension 


WUk's X or F 


P 


Economics Subdomains 


.5580 


.000 


Curriculum Dimension* 


44.24* 


.000 


Curriculum Accent* 


00.13* 


.720 


Node Relation 


.7777 


.000 


Behavioral 


.8891 


.000 


Content 


.o6(U 


.000 


Epistemologicai 


.7474 


.006 


Information Level* 


00.00* 


.984 


Number of Propositions 


"T9798 


.iol 


Representation Level 


.8293 | .000 



9.2 The differences) between students with high and low levels of overall expertise. 

The overall test score of students with low and high level', of expertise can easily be compared. The student 
5jroup can be subdivided into two groups, consisting e.g. of 50% (or 25%) of the students with scores lower 
than the mean (Low expertise LE) and a group with 50% (or 25%) of the students with scores higher than 
the mean score (High expertise HE). It is obvious that - when comparing these subpopulations - significant 
differences are observed in the overall test scores, certainly when taking the extreme quartiles into account. 
However, this is not that obvious when looking at the middle quartiles of the sample as we did. A more 
refined analysis of the test scores, making use of the ten knowledge profile dimensions can help to clarify in 
a much more specific way the specific small or large differences between students with low and high levels 
of expertise. 

Making use of the data gathered among students, studying at the Dutch Open university and at the 
University of Maastricht, a comparable analysis as described in part 9.1 was executed Some problems were 
encountered in relation to basic statistical assumptions about the research data when splitting up the 
population into two groups consisting of 50%-50%. Therefore, only the data gathered among the research 
population consisting of 25% of the students with mean % scores higher and 25% of the students with mean 
% scores lower than the mean were used in the profile analysis. 

All the knowledge profile dimensions help to differentiate - following the results of the univariate analysis of 
variance - between LE and HE students. All differences between both student populations are significant, 
whether we consider the 25%-25% or the 50%-5Q% group. The reader might conclude that the dimensions 
have not been helpful to detect very specific contrasts between LE and HE students. But, the sp-wification of 
the 25%-25% group has proven to be useful since more distinctive differences between low and high 
performers could be detected than comparing the 50%-50% group. Expectations about the knowledge 
profiles - based on our theoretical considerations - have been largely confirmed (e.g. hierarchical nature, 
subsequent difficulty levels, etc.). 

The results of the profile analysis (parallelism test and discriminant analysis) are shown in table 12. 
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Table 12 : Results of the parallelism test in profile analysis 





Wuks A or F 


Pxorpt, 


H^AflAftlf^a niHAAfnaifia 
ECOUUIUIvB VUDOOIUBluS 


.5*0455 


.232 


vui i iw luum love* 




.000 


\+%u iivujuui flvvcm 


536.17 


.003 


nouc cciauou 


.yootu 


.041 


OviMVlOIU ICVCI 




.659 


Content level 


.97667 


.140 


Epistemologtcal 


.24876 


.000 


Number of propositions 


.79462 


.000 


Information level* 


53.55* 


.000 


Representation level 


.95832 


.006 



The results help to detect specific significant differences in the following knowledge profiles of LE and HE 
students : curriculum level, curriculum accent, behavioral level, number of propositions, information level 
and representation level. The results of the flatness test indicate that all dimensions result in non-flat 
knowledge profiles. This implies that all dimensions are helpful to identify a specific structure in the 
mastery of expertise. Following this structure, the mastery of certain components of expertise is better than 
for other components. 



9.3 The differences) between economics and law students, studying the same course. 

"Diploma type" has been an independent variable used regularly in our research (Dochy & Bouwens, 
1990b; Dochy, Valcke & Wagemans 1991b; Dochy, 1992). This research body was helpful to conclude that 
"diploma type" is not useful as a relevant indicator of expertise since conflicting results are obtained from 
the consecutive studies. But the question endures in the context of this text about knowledge profiles. Are 
economics (ES) and law students (LS) for instance - notwithstanding non-significant differences in overall 
expertise - different when looking at their knowledge profiles? 

In the actual analysis, the test results of law students and economics students, studying at the Dutch Open 
university were used for further analysis. These students have subscribed for the course "Economics & 
Money" or "Balance sheet, Profit and Loss Account and Administrative Procedures" at the Dutch Open 
University. The two courses are part of the compulsory program of different diploma lines, such as "Dutch 
Law" and "Economics". 

The results of the univariate analysis of variance when using the knowledge profiles dimensions are not 
helpful to identify more specific and significant contrasts between both student-groups. This affirms our 
earlier research findings showing that "student type" is not a relevant "indicator" of expertise (Dochy & 
Valcke, 1991; Valcke & Dochy, 1991). The initial non-significant difference between both sub-populations 
(overall economics-score) are confirmed, but also the general trend that ES perform better than LS can be 
repeated. 

The results of the multivariate analysis of variance (table 13) are also not helpful to detect specific 
significant differences between ES and LS. The profiles of ES and LS are parallel. The intermediate 
conclusion of non-significant differences between ES and LS, based on the univariate analysis cannot be 
revisited by a more refined analysis, based on the 10 dimensions. The only, slightly significant F-value is 
obtained in relation to the content-level dimension (p=.025). 

When looking at the different components of expertise, as resulting from the flatness test (see Dochy, 
1992), all dimensions/parameters help to differentiate in the mastery of components of expertise - with the 
exception of the "curriculum accent" and "behavioral level" dimension. 
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Table 13 : Results of the parallelism test in profile analysis 





Wilts X or F 


ik or On 

rX "* rP 


Economics subdomsins 


.81048 


.061 


Curriculum* 


3.03* 


.086 


Curriculum Accent* 


1.78* 


.187 


liUUv IwUIUUU 


.93389 


.288 




.98878 


.659 


Content 


.85856 


.025 


Epistemological 


.97850 


.811 


Number of propositions 


.97880 


.453 


Information level* 


1.67* 


.201 


Representation level 


.93829 


.197 



10. Genera! conclusions 



In this report we have introduced knowledge profiles as being graphs of scores of a group or individuals on 
a prior knowledge state test. We defined several dimensions on which knowledge profiles can be based and 
tried to look beyond the subject-matter level. To validate the different knowledge profile dimensions, the 
parameters have been related to the knowledge state test - items and we analyzed the extent to which the 
parameters along the dimensions give information about the components of the PKS. The non-hierarchical 
dimensions suggest that they do relate to different components of the prior knowledge state. This is further 
supported by the profile analyses (flatness tests) when comparing different populations. The other five 
dimensions reveal to be indeed hierarchical, although to a different extent. 

Further, we have been searching for other evidence to validate our 'knowledge profiles' by looking at the 
discriminatory power of the dimensions to detect the PKS differences between several student populations. 
These analyses will be reported more extensively in Dochy and Valcke (1992), Valcke and Dochy (1992) 
and Wagemans, Valcke and Dochy (1992). Although we did not expect differences between ES and LS as a 
results of the ANOVA, this could not be revisited by a more refined profile analysis. ES and LS do not 
significantly differ, but they differ in the mastery of components of expertise, as revealed from the flatness 
test. UL en OU students do not differ in their mean % score, but profile analyses could reveal significant 
differences on the knowledge profiles. Also for students of the LPKS and HPKS groups (middle quartiles), 
differences are found in their knowledge profiles. 

Promising knowledge profile dimensions for differentiating seem to be certainly curriculum level dimension, 
representation level dimension, epistemological dimension, content dimension, economics subdomains 
dimension, and further the behavioral dimension, curriculum accent dimension, information level dimension 
and the number of propositions dimension. Nevertheless, their value is different relying on tae group or in 
this case the goal of the comparison between groups. 

This report is of importance since we succeeded in defining and operationalizing a new and promising 
approach towards the analysis of prior knowledge. It is foreseen that in situations where there are significant 
differences between the PKS of specific subpopulations, the profile dimensions are helpful to detect and 
dissect the strengths and weaknesses of the students involved. This might be a promising starting point for 
differentiated diagnostic and guidance approaches. 
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